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Large-Scale Complex Engineered Systems (LSCES) dominate the aerospace, maritime 
and nuclear industries. Systems such as aircraft carriers, nuclear power plants, spacecraft, 
and submarines represent several unique challenges for the engineering designer, as well as 
the systems engineer. These challenges include extraordinary costs and risks, and the 
inability to fully test and evaluate the complete system until it is nearly operational. These 
systems have become increasingly complex and sophisticated over the past fifty years, 
largely due to the sheer magnitude of inherent couplings that exist between disciplines, 
components, geographically-distributed organizations, and people. These systems are 
regularly prone to crippling time and cost overruns, largely due to the unintended 
consequences arising from unknown or unexpected interactions. The methods, processes and 
tools used by practitioners have not kept pace with the growing complexity of LSCES. 
Indeed, a complex system is now required in order to design and produce a complex system, 
where the elements of both draw just as heavily from social fields as they do from technical 
fields. This paper will explore the challenges that exist in the design of LSCES, the existing 
foundations from which we can draw to build a new framework for design of LSCES, and 
the research opportunities that will hopefully lead to new theories for the design of LSCES. 
This paper represents the opinions and vision of the authors, who recognize that the true 
challenge of advancing the design of LSCES will depend on the views and contributions of 
an extremely broad and diverse number of disciplines from technical and social fields. 


I. Introduction 

E NGINEERING design has enabled many great feats in the course of human history. Recently, we were able to 
design a system to land men on the moon just over 40 years ago. Today the computers in our phones are more 
capable than the computers were within the Apollo capsule. We can land an aircraft in near zero visibility - safely 
and repeatedly. Submarines use nuclear power to help them remain submerged for months. So, given these 
tremendous achievements, why are we still talking about challenges in designing large-scale engineered systems 
today? Is it conceivable that future large-scale complex engineered systems may be designed by continuing to do 
more of what we already know how to do today using more advanced technologies developed tomorrow? In the 
opinion of the authors, in many cases, the answer is no. Why is this? 

Large-scale Complex Engineered Systems (LSCES) include aerospace (e.g., aircraft, space systems), large 
maritime (e.g., submarines, aircraft carriers), nuclear (e.g., power plants), and major civil infrastructure systems 
(e.g., water supply systems, electric power grids, offshore oilrigs, and air and ground transportation systems). The 
design of such systems has enabled capabilities that have transformed the way of life in much of the 
world. However, the cost and time to develop these systems are growing at an unsustainable rate. And, in many 
cases, discontinuing development of one of these systems may be not an option. We rely on LSCES to meet many 
basic local and national needs, such as provision of essential services, protection from natural and human dangers, 
preservation of resources, and advancement of infrastructure capabilities to address population growth. 

Though failures of LSCES are not common, yet they may occur even when best practices have been used. And, 
failure of most of these systems is not tolerable due to the often widespread impact of LSCES failures with local to 
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international implications. Further, many LSCES are connected to other critical systems, large and small, such that a 
failure of one system has an impact on a myriad of other systems. For example, an electrical system black-out may 
impact transportation systems, medical services, banking, communication networks, etc. Likewise, failures (or even 
inefficiencies) in transportation systems can adversely affect many business networks such as food supply. 

For most industrialized nations, LSCES are critical, costly, cannot fail, are connected to other systems, and take 
several years to develop. The science of the design of (and likely the operation of) LSCES is being pressed to evolve 
to address the growing demands placed on these engineered systems. Growing human needs in national defense, 
environmental sustainability, medical advancement, and human prosperity continue to be drivers for the increased 
complexity seen in engineered systems. While technology and these growing needs have pushed us towards ever- 
more complex products and systems at an extremely fast rate, some of the methods, processes and tools we use to 
conceive and bring these systems and products to fruition have not kept pace 1 " 2 3 . In this paper, we explore some of 
the present challenges of designing LSCES and discuss a few areas of future promise. We begin with an overview of 
the characteristics of this field of engineering, followed by a discussion on the challenges of designing these 
systems. We conclude with some perspectives on areas of research that may improve future LSCES design, which 
draw from engineering as well as social science fields of study. Mostly, we hope to inspire conversation and spur 
new avenues of research pertaining to these systems and their design. We believe increased awareness of 
researchers, who hold diverse expertise and experience, represent the best foundations and the greatest promise for 
advancing the design of LSCES in the future. 

II. The Nature of Large-scale Complex Engineered Systems 

LSCES include a diverse array of large systems, from airplanes to nuclear power plants (e.g. Fig. 1), yet a few 
salient features characterize many of these systems, including: 

• Significant cost and risk; 

• Extensive design cycle; 

• Protracted operational timeline; 

• Complexity; and 

• Large, dispersed organizations. 

Each of these characteristics is discussed below. 

LSCES costs are typically in the tens of millions to billions of dollars for one system. These systems tend to 
operate in highly unforgiving and often uncontrollable environments such as oceans, outer space, or those using 
substantial quantities of toxic chemicals. Failures in this field of engineering tend to have local, national, or 
international impacts that can affect economies and national defense. Failures also have the potential for significant 
loss of human life. Mistakes during design might result in millions of dollars or years of delays in fielding a 
system. As such, governments, politics, and, often, national defense are inherently involved even for private or 
commercial ventures 4 . 

The high levels of risk, in terms of costs and impacts of technical failure, have substantial implications on the 
extensive design cycle of LSCES. The nature of LSCES makes it often too difficult, risky, expensive, or impossible 
to use conventional trial-and-error engineering methods. Often, the final “prototype” is the first and only complete 
system trial. Additionally, many new LSCES are often one-of-a-kind systems with little to no empirical data; hence 
some experimentation (with constraints on full-scale prototyping) is inevitable during design, as complete system 
analytical models often have some deficiencies. Though designers generally try to focus on derivative solutions to 
reduce unknown-unknowns in developing the system, unknown-unknowns persist, even with derivative 
systems. Simply scaling up or adapting previous systems may not be an option available for some LSCES designs 
such as the Next Generation Air Transportation System 5 . 

Though traditionally design is said to start with the release of near final system requirements and ends once a 
system concept is envisioned, in reality, the design cycle for LSCES begins in the laboratory and ends when 
production begins. Most new LSCES push the performance envelopes of existing systems and thus rely on some 
advanced technologies, some of which may not be “commercial off the shelf.” Hence, the decisions of researchers 
that are developing advanced technologies for the LSCES (such as new materials or electronics) impact the system 
design as the new technologies are incorporated into the emerging concept. As full-scale trial and error methods are 
compromised for LSCES, new information from sub-component tests continues to update the design, often until 
production begins. The next section further delineates many of the processes for the design of LSCES today. 

LSCES are generally in use for several decades. Note that this protracted operational timeline (as compared with 
other products such as automobiles or computer systems) means that LSCES designs are often challenged to adapt to 
changing political and societal needs. For example, LSCES designed today will likely not have the energy efficiency 
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or computing capabilities that are likely to be desired of systems in 10 years, yet the lifespan of a LSCES may be 30 
or more years. The pace of technology advancement today is outpacing the ability of some LSCES designs to adapt 
as needed. 

Further, most LSCES must interact with another LSCES with high-degrees of precision. This is the case, for 
example, for: 

• Fighter aircraft on an aircraft carrier; 

• Commercial aircraft in the air traffic control system; 

• Nuclear power plants within submarines; 

• Space vehicles docking at the international space station; and, 

• Multiple systems (capsules, rocket engines, etc.) interacting to launch major payloads into space. 



Figure 1. Representative large-scale complex engineered systems 


Thus, LSCES designs must keenly address interoperability so that one system may work with existing and 
changing infrastructures. For example, aircraft carriers designed more than a decade ago must be able to adapt to the 
aircraft of today and tomorrow. Electrical grids designed today need to consider how the grid may be adapted during 
use to address very rapidly growing electrical power needs of the future. 

Often, LSCES are meticulously and cautiously modified throughout their lifespan to adjust to new findings and 
changing demands. However, the complexity of the system makes each design modification a nontrivial effort. As 
with all complex systems, a change to one part of the system can propagate to distant parts of the system, sometimes 
in a manner that is extremely difficult, if not impossible, to trace. High-fidelity simulation models are often 
extensively used to thoroughly “test” system upgrades. Though these simulation “testing” methods are largely 
sufficient (especially when combined with smaller-scale physical tests), at times, impacts of system upgrades are not 
fully apparent until the system is in use. 

The term “complex” in defining LSCES stirs much debate, for good reason. Although complexity science has 
now been researched for over 25 years, opinions as to what is truly complex (versus complicated) vary wildly in the 
literature. Further, the terms complex and complicated are often used interchangeably throughout the literature. 
While existing literature does not provide much quantitative consistency in conceptualizing complexity, the basic 
tenets of complexity are largely accepted and are relevant to LSCES, as they distinguish LSCES from more common 
complicated systems. 

Whereas complicated systems (such as a computer or television) may be more given to classical reductionist 
approaches of development where the system may be reduced to its basic elements or parts and then studied largely 
as a summation of its parts, complex systems greatly strain or even nullify reductionist approaches 6 . In a complex 
system, the interdependent sub-elements exhibit dynamic and reflexive intra-system behavior. 

We define complex systems to be those for which tightly coupled interacting phenomena yield a collective 
behavior that cannot be derived by the simple summation of the behavior of the parts. In essence, these are highly 
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interdisciplinary systems in which the existence of inherent, and often unknown, couplings potentially leads to 
irrational results. Complex systems may be biological (human body), natural (rain forests), or engineered (some 
medical devices, genetic algorithms, and neural networks). LSCES, by themselves, may or may not be complex, 
depending on their design. Many LSCES exhibit the characteristics of complexity defined previously, particularly in 
terms of pervasive and unknown couplings and behavior as a singular integrated apparatus that is often beyond a 
modularized network of pieces. 

However, as all LSCES are inherently designed, operated, and used by humans, the combined engineered system 
with the social system that interacts with it creates a complex system (what some call a “socio-technical” system). 
An example of this is the emergent nature of learning. As new data (and hence knowledge) is acquired during 
system development, the system design is updated in a manner that is not always predictable a priori. The learning, 
and corresponding system updates, may also occur in an inconsistent manner across the many sub-systems. 
Managing the impacts of new findings in one sub-system on other, coupled sub-systems has long been a challenge 
during LSCES development and is often one of the many reasons for unpredictable schedule and cost over-runs 
during development. 



Figure 2. Exploring Complex Functions Using Domain Coloring 


The size and scale of these systems results in design teams generally exceeding a thousand engineers, with the 
number of participants across all activities being potentially in the tens of thousands. As an example, consider the 
Boeing Company and its design team for an aircraft. The Boeing Company has over 170,000 employees in 70 
countries, 140,000 of who hold degrees (approximately 35,000 advanced degrees) 7 with employees organized in 
3500 teams 8 . Boeing draws upon the expertise of hundreds of thousands additional people working for Boeing 
suppliers around the world. The development of the Boeing 787 Dreamliner engaged a significant number of its 
employees as well as over 50 supply partners, who were fully engaged in the development process from the 
beginning of the detail design phase. The teams for this development project were distributed across 135 sites 
around the world. The development program was launched in April of 2004 and delivery of the first aircraft was 
made in September of 201 1 . 9 

As described above, a typical aspect of LSCES design is that the participants are dispersed across geographical 
locations and multiple organizational entities (contractors, government laboratories, universities, etc.) resulting in 
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many system “team” members never personally interacting with large portions of the LSCES system “team” and 
each entity having its own (often very different) incentive system. These individuals and smaller teams are exposed 
to political, societal, and organizational forces and demands that impact the design formulation and process in often 
subtle and unexpected ways. For example, a sub-contractor may be driven to use certain types of mechanical parts to 
help its local economy and business growth, even if these parts may not be the best parts for the overall system. 
Every design team involved controls multiple attributes of their own piece of the design, which will ultimately 
impact the system level attributes. 

The impacts of the divergent preferences within the design team on the ultimate system concept are significant 
and often overlooked. This is discussed in greater detail in the next section on design. Regardless of the widely 
varying forces impacting the dispersed team developing a LSCES, in aggregate, the highly interdependent design 
“team” has to attempt to operate as synergistically as possible - closer to that of a symphony (or a living organism) 
of many diverse but intertwined, interdependent parts - rather than an assembly line (or mosaic) of diverse but 
joined, independent parts. The former (operating synergistically, similar to a symphony) is increasingly challenging 
for LSCES teams as the engineered systems continue to grow and correspondingly, the organizations continue to 
increase and leadership demands (particularly during systems engineering) escalate. 

Considering a wide view of the total system, the resulting 
“system of systems” involved with LSCES is extensive: a large 
engineered system that is made up of many smaller engineered 
systems (one system of systems) that is designed, developed, 
and operated by another large “system” of dispersed, loosely 
connected people (another, constantly emerging, system of 
systems), and this large-scale, complex socio-technical system 
of systems (a LSCES) must interact with other large and small 
systems that may also have human interactions. The resulting 
interactions defy easy capture or representation, with 
interactions and interfaces being critical, along with the type of 
interface, the importance (e.g. strength) of the interface, and 
the quality of the interface (see Fig. 2 10 ). As noted previously, 
many LSCES are critical to major infrastructures and 
interdependent on other systems, creating massive systems of 
systems that must be carefully designed and operated. 

In the next sections of this paper, we attempt to summarize some of the multi-faceted challenges with designing 
LSCES and highlight some of the opportunities. The engagement of several research communities including 
engineering as well as many in the social sciences, is necessary in this discussion. We then present summaries of a 
few contributing fields of research. Fundamentally, we propose a study of the science of designing large-scale 
complex engineered systems that incorporates more of the factors that influence system design outcomes then 
presently exists. These include: government contracting, organizational communication approaches, decision 
making, modeling and simulation, economics, conflicting incentive systems, systems engineering processes, 
leadership and management skills, incorporating new technologies during design, facilitating creativity or 
cohesiveness in the dispersed technical organization, policies and politics, etc. One view of some of the contributing 
factors is shown in Fig. 3. Yet, as the research community of the science of designing LSCES grows, improvements 
to this Venn diagram are needed and welcome. 

III. The Design of Large-scale Complex Engineered Systems 

As evidenced by the data pertaining to the impact of cost and time overruns previously discussed, the methods, 
process and tools used for traditional product design are simply not appropriate for the design of large-scale complex 
systems. Some of the innovative approaches used at Apple or IDEO cannot be adequately scaled for systems as 
complex as the Next Generation Air Traffic System (NextGen) or even the next Boeing aircraft development project 
that will engage thousands of participants in the process - most likely from numerous companies scattered across the 
globe. This brings up questions of how the design process differs for these systems, what its relationship is to 
systems engineering, and the time scale at which design occurs for these systems. 

A. Systems Engineering and Design: Present Practice and Future Potential 

Today, large-scale complex systems are designed within the context of a larger Systems Engineering process. 
The International Council on Systems Engineering (INCOSE) defines Systems Engineering as “...an 
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interdisciplinary approach and means to enable the realization of successful systems. It focuses on defining 
customer needs and required functionality early in the development cycle, documenting requirements, then 
proceeding with design synthesis and system validation while considering the complete problem... 11 ” The complete 
problem, according to the INCOSE definition, includes: operations, performance, test, manufacturing, cost & 

schedule, training & support, and disposal. The definition continues to say that SE “integrates all the disciplines and 
specialty groups into a team effort forming a structured development process that proceeds from concept to 
production to operation.” It is clear from this definition that design is considered to be one of many sequential 
activities placed within the structured (i.e. hierarchical) development process. Today, the most common system 
engineering approaches involve using a hierarchical decomposition within a requirements-driven framework, as seen 
in Fig. 4 12 . The research, development and design processes are often decomposed into smaller, more manageable 
sub-systems that are often functional (wing, engines, etc.) or disciplinary (structures, aerodynamic, etc.) in nature. 
Each decomposition, however, creates artificial boundaries and couplings that then need to be managed during 
systems engineering through interface control documents. 



Time 

Figure 4. Systems Engineering “V” Chart 


Alternatively, Michael Griffin, former NASA Administrator, describes the primary responsibility of systems 
engineering as “... the fielding of an elegant design,” where an elegant design is defined to be “one which produces 
the intended result, is both robust and efficient, and generates a minimum of unintended consequences 2 .” In this 
paper, we define design as the process of making rational decisions regarding available alternatives in order to 
achieve one’s stated preference. The stated preference, if we embrace Griffin’s definition of systems engineering, is 
the achievement of elegance in the system, where elegance is defined by the four attributes above. 

These differences in definition produce vastly different approaches to the design and development process for 
large-scale complex engineered systems. The hierarchical decomposition used in SE processes is based on the 
inherent assumption that the sum of the parts will be equivalent to the whole. However, the reality for LSCES is 
that the inherent interactions and interdependencies between disciplines and subsystems make it very difficult, or 
likely impossible, to understand how the modifications in one remote part of the system will impact every other part, 
regardless of the number of Interface Control Documents (ICDs) produced for the system. Hence, it becomes clear 
that existing processes alone will never be enough to overcome the time and cost challenges presently being 
experienced during development. More is required. What is required is a science pertaining to the Design of 
Complex Engineered Systems. Given that we fully embrace Griffin’s definition of Systems Engineering, then what 
is required is a Science of Systems Engineering that takes a holist view of the problem space from basic research to 
operations. But, what would such a science look like? And what is the context in which systems design would be 
implemented across all aspects of the development cycle? 
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B. Design in the context of Large-scale Complex Engineered Systems 

Achieving an elegant system design has significant and far-reaching implications - not only for the definition of 
systems engineering, but also for the understanding of what design is and the timespan over which design activities 
occur for these systems. One aspect of the nature of these systems that has significant design impact is that they are 
typically one-of-a-kind systems. As a result, new technologies, methods and processes often need to be developed 
concurrent with the development process itself. New and disruptive technologies demand that research and 
development, and system design phases be integrated in a seamless fashion, resulting in greater iteration across the 
lifecycle of the complex system. Research, systems engineering and design cannot be separate functions (as they 
often exist today), if the system is to ultimately be elegant. Why is this? 

Today, researchers engage in fundamental research activities, publish papers identifying their research activities 
and outcomes, and then move on to the next research project, assuming that someone down the line will pick up 
their idea and develop it as needed into a workable or useable technology. However, given that many of the 
independently developed products from research will work interdependently together in the actual large-scale 
system, what information, knowledge, or systems impact is lost in researching and developing the interdependent 
components separately? Further, as mentioned in the previous section, design is considered to be an activity that 
comes after requirements are identified and the system is hierarchically decomposed. Once the design tasks are 
complete, production commences, with system operation following. These are not presently considered to be 
interrelated tasks, although it is clear that achieving an elegant system design that works as intended , is robust , 
efficient and has a minimum of unintended consequences , would actually require engagement across the lifecycle of 
the product, as well as reaching back to research stages. For complex engineered systems, design quite often needs 
to begin in the laboratory, as pressing environmental, economical, and policy challenges must be addressed from the 
earliest stages of research. 

For a system design to work as intended , there is a need for operational issues to be considered from the outset. 
How the ultimate end-user will interact with the system once operational is critically important in the designer’s 
decision-making process. More fundamentally, there is a need for a clear understanding of the intent of the system. 
What is it that the system is really being designed to do or accomplish? Such understanding may, itself, require 
significant iteration across constituencies as well as stages in the development process. For the system to work as 
intended, manufacturing, operation and disposal becomes just as critical as traditional performance objectives. 
Enabling this type of integration, which is also heavily dependent on the people involved in each facet of the 
process, as well as the organizational structure, represents significant opportunities for future research. 

For a system design to be robust, there is a need for appropriate technologies to be incorporated in the design, 
with an awareness of how all stages of the development process might impact robustness. For instance, small 
variations in environment, operations, manufacturing, or other unrepresented interactions, including those resulting 
from the people and organizations involved, can produce unanticipated or ‘emergent’ behavioral responses. Hence, 
the design team is challenged to represent the uncertainties associated with both earlier and later phases of the 
system’s life cycle. A full engagement of the design team in all stages of the development would certainly be 
challenging, but likely extremely worthwhile. Numerous research questions can be posed regarding robustness in a 
systems context. 

For a system design to be efficient , a mechanism to distinguish amongst competing alternatives needs to exist so 
as to be able to determine the best choice (i.e. that system design which best satisfies the desired preference with a 
minimum expenditure of resources). Again, this requires input well into the operational phase, and even into 
disposal. A system whose cost skyrockets at end of life due to environmental or policy reasons would most likely 
not be an efficient solution. Additionally, however, is the need to engage the design team in the definition of the 
system objective (i.e. preference) from the outset of the process in order to be able to effectively define efficiency. 

For a system design to have the minimum number of unintended consequences , a full understanding and 
appreciation of inherent couplings is critical (which includes an understanding of how couplings will impact the 
system level preference), together with a design process that embraces these couplings and accounts for them 
effectively. This also means that a decomposition performed separately from design considerations might not be the 
best approach. Decomposition for LSCES, so necessary for management and implementation reasons, could become 
an integral part of the decision making process. The means by which this might occur provides significant 
challenges and offers rich possibilities for research. Additionally, a full understanding of the impact of people and 
organizations on the system would help in the creation and management of design teams, another rich area for future 
research. 
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C. Emergent Properties and Design of Complex Engineered Systems 

The impact of humans in the design loop inherently creates emergent properties. As noted earlier, learning 
naturally creates emergence as parts of the design are updated throughout the development process. Another source 
of emergence comes from divergent preferences in the design team. The scale of the systems being designed lend 
themselves to situations in which the participating teams and the individuals within the teams often have divergent 
preferences. The present systems engineering approach, in which requirements are passed down from level to level 
in the hierarchy, often makes it challenging for systems level issues to be understood across these levels. Another 
major challenge in the design of complex engineered systems, therefore, is to minimize the inevitable irrationality 
that results from large groups of people working together in distributed environments. Further, it is extremely 
challenging for design teams at the lowest levels of component design to adequately appreciate or understand the 
objectives (and impact of their design decisions) at higher levels. The difficulty of individuals or participating teams 
to understand the entire system and appreciate all the existing interactions often leads to challenges in effectively 
communicating system objectives, tradeoffs, and risks across teams. Rational decision-making, a critical component 
of design, can become seriously compromised in such a setting. These circumstances lead to emergent behavior, 
every bit as serious as that resulting from uncaptured couplings in the physics. Hence, the social and cultural issues 
become just as important to model and capture in the design process as the mechanical physical modeling. 

D. Design Science for Large-scale Complex Engineered Systems 

The widespread and rapidly growing prevalence of complex engineered systems means that the critical gaps in 
system design theory and methodology have pervasive and potentially damaging impacts from small to large scale. 
Given that we do not sufficiently understand the nature of these LSCESs, we do not yet have at our disposal all the 
fundamental rigorous methods, tools and processes that will allow us to effectively, efficiently and efficaciously 
design them. We believe there is a driving need for a theory of Design for LSCES, which would include not only the 
physics based disciplines, but also the organization and human psychology aspects of design that are inherent to the 
process. The need for a design science, spanning small to large-scale complex engineered systems, seems clear and 
compelling. The hallmark of such a science, according to Thagard 13 is that it would: 1) use correlative thinking (e.g. 
A regularly follows B in controlled experiments); 2) seeks empirical confirmations and disconfirmations; 3) 
practitioners care about evaluating theories in relation to alternative theories; 4) uses highly consilient (i.e. explains 
many facts) and simple theories; and 5) progresses over time, developing new theories that explain new facts. We 
believe there are tremendous opportunities for building upon existing foundations across numerous disciplines of 
study to move forward in developing a new framework - and science - for DLSCES, built upon solid theory and 
rigorous mathematics. 

IV. Future Directions for the Design of Large-Scale Complex Engineered Systems 

As noted earlier, we propose the development of a science of design for LSCES that incorporates more of the 
factors that influence system design outcomes. We recognize the problem space is considerably larger than has been 
fully conceptualized in the research communities to date. While current efforts to find solutions to the persistent 
problems of schedule and cost overruns, missed performance targets, and occasional but catastrophic failures have 
made significant progress, we recognize that we have yet to adequately address many areas of the problem space. 
And, there are many areas of the problem space of which we are likely still unaware. A question we pose to the 
broader research community is: What is missing? 

Theories related to decision analysis, game theory, and MDO have been the focus of much of the research on 
advancing the design of LSCES. As a discipline of practice, the systems engineering community has also necessarily 
begun to treat this practice as an area of research. The field of complexity science has begun to focus more on 
engineered systems, with a growing body of literature in this area. And, concepts and theories from psychology and 
organization science have also begun to be considered in advancing the science of design of LSCES 14 . 

As we look forward toward the needs and opportunities for the science of design for LSCES, we recognize great 
disparity in the contributing communities of research in terms of awareness of key problems, maturity, applicability, 
and connection with practitioners. Some research communities, such as MDO, have been specifically working 
toward many of the challenges for LSCES for several decades and have been well engaged with practitioners. Other 
research communities of interest, such as law and associated contracting practices, have not been sufficiently 
engaged in identifying opportunities for improvements for the design of LSCES, though their research is of high 
relevance. Other potential areas of research that may add considerably to advancing this topic, but have yet to be 
adequately addressed, include policy and political science, education and training, business operations and logistics, 
multi-cultural challenges in global design teams, and many others. 
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The needs of designing LSCES are many and diverse, particularly when considering the integration of social and 
technical issues. In this section, we will touch on a few areas of research for the design for LSCES and identify some 
of the relevant theories and disciplines that have promise in contributing to the field. Each of these areas of research 
have bounds of applicability to LSCES, as well as areas of interrelation with other research. Here we focus our 
discussion on four (of many) pressing needs for the DLSCES. The four areas present opportunities that could have 
significant impact in the near-term, if more extensive research were conducted in these areas using existing research. 
Each of the four needs identified below has both technological as well as social aspects, resulting in engineering and 
human challenges that must be simultaneously considered. 

A. Four Key Needs Representing Research Opportunities 

1. Defining the System Objective: This relates to the need to correctly identify and represent the desired 
objective or purpose of the system being designed, with a full understanding of how the necessary 
resources required to create that system, as well as the risks associated with it, will be influenced by the 
identification and representation chosen. 

2. The Inherent Interactions in LSCESs: There is a pressing need to fully understand, manage, and exploit the 
inherent interactions in the system (from people, organizations and the physics), in a rigorous manner 
grounded in theory, so as to avoid unanticipated consequences during the design and development process. 
One outcome of this need is understanding how organizational structure (which determines communication 
flow, incentive structures, etc.) might impact the final system design. 

3. Communicating the System Objective: There is a pressing need to be able to effectively communicate the 
desired system objectives to all participants in a way that enables the impact of their decisions (regardless 
of how seemingly small and insignificant their role or contribution is to the larger system) to be fully 
understood in the context of the system. At present, the requirements driven approach to design of these 
complex engineered systems makes it challenging for all teams and individuals involved to understand how 
their decisions will impact the overall system objective. 

4. Decision-making and Sense-making with Ambiguity and Uncertainty in LSCESs: At present, there is an 
extreme need to better capture, understand and use the collective, synergistic, and tacit knowledge of the 
organizations involved to produce more efficient, effective, and reliable systems, while appropriately 
capturing and modeling the resulting uncertainties meaningfully, across the entire engineering spectrum 
and across all participants. However, before uncertainties can be captured, the ambiguities must first be 
addressed. These are not mutually exclusive steps, but are rather integral to one another. 

Each of these needs will be explored in the following sections, highlighting existing theories and methods from 
engineering, social, and organizational sciences that have potential to impact the design of LSCES ’s, together with 
research opportunities associated with each need. We again point out that these are only a small subset of needs to 
be addressed in this very rich and emerging field of study. 

B. Defining the System Objective 

While this may seem trivial, it is perhaps one of the most challenging parts of the LSCES design and 
development process. In the article 15 , “Don’t Come to the Dark Side”, a DoD acquisitions branch chief opined that, 
while the Emperor wanted a Death Star, was it really a good idea and was it really what he wanted? Would the 
Emperor have wanted the Death Star if he knew that a well-placed laser shot into an exhaust vent by a half-trained 
Jedi in a beat-up X-wing could cause the entire thing to blow up? The Emperor may have thought he wanted a 
Death Star, but what he really wanted was universe-wide supremacy. He mistakenly thought the Death Star would 
be the means by which his objective would be satisfied. He was unfortunately (for him) incorrect. Had the Emperor 
instead framed his desired preference correctly (i.e. to be Ruler of the Universe), the resulting system that would 
enable him to achieve his preference would have been something vastly different from the Death Star. This is 
patently obvious since the Death Star ended up facilitating his demise. While this may seem tongue-in-cheek, it is a 
fundamental issue that represents challenges across Federal agencies and private companies on a regular basis. 

Consider the case of the U.S. Army’s SLAMRAAM anti-aircraft missile system (prototype in Fig. 5), which was 
finally cancelled after ten years and approximately $3 billion spent on development 16, 17 . The system would have 
cost another $12 billion to put in production. The issue, however, was that no one really wanted the SLAMRAAM. 
There were no orders for it. A major reason was that it wasn’t a unique design - there were very effective systems 
previously used (the Chaparral and then the Avenger). The Army thought they wanted SLAMRAAM, which would 
use larger air-to-air missiles, but what they actually needed was a “shorter range system that could deal with enemy 
helicopters.” 
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Figure 5. Surface Launched Advanced Medium Range Air-to-Air Missile Launcher Prototype 


The basic issue is in the initial framing of the objective (expressed as a preference in decision science and part of 
the problem formulation challenge). A private company might frame their objective as designing and building an 
aircraft with 10% greater capacity than any existing vehicle in order to corner the market in overseas civilian 
transport. Such a stated objective might actually negatively impact the company, however, since what they might 
really want is to maximize their profit over a specified term. These two preferences, communicated to the design and 
development teams, would result in vastly different systems. 

The appropriate identification of the system objective is critical so that the outcomes of all the available 
alternatives being considered during the design and development process can be compared against the stated 
preference. This process, rooted in Decision Analysis, is addressed later in section 4. It is also critically important 
for any mathematically based engineering method that might be used in the process (e.g. optimization), since the 
choice of objective function will dictate the final design (when optimization is used). The implications for choosing 
the wrong objective (i.e. one which doesn’t actually represent what your true preferences are) are discussed in 
Hazelrigg 18 . 

Returning to the concept of systems engineering being the fielding of an elegant design, we recall that one of the 
attributes of elegance is that the system work as intended. We offer an extension of this requirement to state that the 
system should be an accurate embodiment of the designer’s preference and should subsequently work as intended. 

The authors believe it is important for there to be an understanding of the ramifications of this step, which feeds 
into every other step of the design of LSCESs. This need represents an opportunity for research in the future, 
drawing heavily from many of the social science topics presented previously. A unique opportunity would be to 
combine the relevant social science theories with those drawn for more traditional engineering areas to answer such 
questions as: What would the ramifications of a particular objective have on procurement? How would procurement 
strategies impact the ability to achieve a particular objective? How would a shifting policy stance impact the 
outcomes associated with one choice of objective over another? What value would one objective bring versus 
another with regards to profit or mission success, to technological readiness, or to organizational risk? 

C. The Inherent Interactions in LSCESs 

In just over a hundred years of air-powered flight, we have moved from a design team of two participants to tens 
of thousands. We’ve gone from very basic representative physical models to extremely high fidelity models. The 
expansion of both the technical and physical modeling, together with the human element, results in a number of 
interactions in the design of a large-scale complex engineered system that is likely in the billions. 

An avenue that has been recognized as providing substantial benefit pertains to the incorporation of more 
sophisticated modeling, simulation and optimization approaches at the earliest stages of design, where trade-offs are 
often just as critical, if not more so, than at the detail stage. The use of optimization for design of these systems has 
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largely been relegated to the detail design stage, and confined largely to subsystems. The same is true of modeling 
and simulation, where the highest fidelity models for component design are often assumed to be best, regardless of 
whether the fidelity is required to achieve system level information. The ability to harness the substantial power 
available in a framework that integrates these physics-based capabilities with those offered by social science would 
be of immense value. The next sections discuss each of the physics and math-based capabilities and their potential 
contributions to DLSCES, particularly in light of opportunities for integration with social science fields. 

1. Modeling and Simulation 

Modeling and simulation is a critical component of DLSCES, and has tremendous potential for aiding in better 
decision-making. The sheer size and scale of LSCESs makes building and testing the system extremely difficult and 
very costly. Build and test approaches for components provides considerable data, yet given the inherent and 
pervasive couplings that exist, creating a true test environment very challenging. Modeling and simulation enables 
the use of virtual prototypes that can capture the physics of each participating discipline so as to enable a virtual 
analysis of how the alternatives being considered compare to the stated preferences. The use of modeling and 
simulation makes design of these systems feasible and can potentially reduce the time and cost associated with the 
development process. As described in Reference 19 , simulation is of particular importance since it provides a means 
for representing the impacts of the inherent couplings in the system. Figure 6 20 , for instance, is a visualization of 
rotor wakes produced during a simulation (by NASA’s Subsonic Rotary Wing Project) to enable more accurate 
prediction of vehicle performance and noise production, by capturing the interactions between the rotor blades and 
rotor blade vertices. 

In Reference 19 , the authors provide an overview 
of the state-of-the-art in modeling and simulation, 
with a focus on design of large-scale systems. 

Effective elements of a system design simulation 
capability include: sufficiently expressive modeling 
languages to model the challenging phenomena 
encountered in such systems; ensuring simulation 
models are easy to create and reuse; and a 
collaborative modeling capability that supports the 
collaborative environment in which these systems are 
designed and developed. 

Ultimately, these models and simulation 
capabilities are used to enable decisions to be made. 

Because we know all models (and therefore 
simulations) have errors, the important question 
becomes one of validity. The general assumption 
that a higher fidelity model will produce better 
answers is quite often invalid in a systems context. All components in a systems model will not impact the system 
preference in the same way. If we think of it in terms of system derivatives, it becomes a bit easier to consider. Let 
us say that the control system analysis produces an output that influences are system objective only in the tenth 
decimal place while our structural analysis produces an output that influences the system objective in the thousands 
place. This suggests an investment strategy in which time and effort in ensuring the structures is more accurately 
modeled than the controls would be best. However, having said this, it still begs the question of how good is good 
enough. Trade-offs in model fidelity for system design have been studied in the past 21 , but remain a rich area for 
further research. A research opportunity here might focus on cultural issues in which teams are resistant to trust data 
from other teams (particular from other organizations or companies), given lack of sufficient providence. Such an 
issue is commonplace in practice, but not well-studied. Related to this would be a question of how to appropriately 
incorporate uncertainty associated with the confidence in the data coming from other teams, which would impact the 
trust associated with models built from that data. Also, how can fidelity trade issues be managed across multiple 
teams in multiple organizations and companies? 

2. Multidisciplinary Design Optimization 

Multidisciplinary Design Optimization (MDO) offers a means to include an objective function at a system level, 
propagate requirements to coupled subsystems rationally, and to model and represent the impact of physical 
subsystem couplings on system level objectives, while still enabling a decomposition of the system to manageable 
proportions. MDO evolved in the 1980s out of structural optimization needs in engineering. Given that MDO is 
heavily dependent upon computer simulation and modeling, the existing computational capabilities significantly 
limited the application of MDO. However, the rapidly growing computational capabilities were sufficient to enable 



Figure 6. V-22 Rotocraft Cross-Section 
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the broadening of optimization criteria beyond structures, including such disciplines as controls and aerodynamics. 
Aerospace companies, agencies and academic researchers recognized that this computational infrastructure would 
support the development of new methods that would capture and represent the inherent couplings that exist between 
disciplines in aerospace vehicles and systems. 

Initial efforts in the field explored bi-level 
hierarchical linear decompositions 22 , but quickly 
grew to encompass, by the late 1980s and early 
1990s, non-hierarchical decompositions 23 . Such non- 
hierarchical representations were made possible by 
the Global Sensitivity Equation (GSE) method 24, 25 , 
which provided a means of quantifying global 
system sensitivities without requiring system level 
finite difference. The inclusion of system level 
couplings, through the GSE sensitivities, provided a 
way for designers to implement an all-in-one 
optimization with a system level objective function 
and constraints (representing requirements) drawn 
from multiple disciplines. The field quickly moved 
beyond this to enable simultaneous optimizations in 
distributed subspaces that still captured the coupling 
information from the different disciplines. The 
evolution of MDO methodologies has largely been 
tied to computational resource availability and 
growth, with ever more complicated and complex 
systems being explored. At the same time, the 
computational advances have continued to spur new 
frameworks and methods in the field. 

MDO has produced a set of design engineering methods that search for the best design of a system within a 
constrained design space, while ensuring that the inherent couplings between disciplines and subsystems (so 
prevalent in large-scale complex systems) are appropriately represented. MDO methods are not limited to 
optimization methods, however, as the field has contributed methods for addressing coupled analysis, system 
optimization, system approximation, visualization, and trade space decisions. An example of a visualization 
technique known as graph morphing representation, a visual design steering method, is seen in Fig. 7. Ultimately, 
the field of MDO is about the development of methodologies to support the analysis and design of multidisciplinary 
systems. Research from the MDO field has tremendous potential to contribute to the design of LSCES. 

At present, the promise of MDO has not been fully realized. One of the challenges is that the present systems 
engineering framework used for large development projects, does not easily lend itself to a system level 
optimization or a non-hierarchic decomposition structure. Present practice in systems engineering is to decompose a 
complex system or product, design each component separately, and then recompose the system, using nterface 
control documents(ICDs) to capture much of the interface information. The field of MDO has assumed, as an 
operating premise, that any decomposition not adequately capturing the inherent interactions will never lead to an 
optimal design. One area of future potential may come from proactively blending the field of MDO into systems 
engineering practices. 

Challenges for MDO research in the context of design of large-scale complex engineered systems includes the 
fact that MDO assumes the objective function is an exogenous input (i.e. the objective has been previously defined). 
The inclusion of finding the ‘right’ objective function as a design goal is a potentially rich area of research. This 
need ties back to the first key need identified for LSCESs - “Defining the System Objective”. Until this step has 
successfully been accomplished, MDO methods cannot be truly effective. Also, MDO has not (until lately 26 ) 
focused on the organization or individual engineers involved in executing the design. Given that MDO was 
originally intended to achieve as much automation as possible, human inclusion was not a priority. Elowever, the 
recognition that MDO methods, processes and tools must put designers ‘back in the loop’, was one of the key 
themes in a recent NSF sponsored workshop exploring the future of MDO in the context of design of complex 
engineered systems 27 . Such inclusion of the designers, as well as the organizational aspects so critical to the eventual 
system, is another rich area for potential research. MDO depends implicitly on modeling and simulation, so many of 
the challenges seen for modeling and simulation can be extended to include MDO. Rich avenues of research might 
include exploration of human-in-the-loop MDO strategies that provide for team and individual preferences to be 



Figure 7. Human-in-the-loop Visual Design Steering. 
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represented for trade space exploration. Questions might include: How can individual preferences be represented 
and modeled in an MDO framework? Can modeling and simulation of individual preferences be used to explore 
team interactions that would benefit the MDO process? How can the inherent individual and team preferences be 
modeled alongside the inherent physical interactions? Can such modeling be used upstream in the development 
process, to simulate later stage interactions that could subsequently be incorporated in an MDO process? These and 
other similar questions could substantially advance the ways in which interactions are understood and managed in 
DLSCES. 

3. Complexity Science 

Complexity Science explores simple causes for complex effects, with an underlying assumption that “complexity 
in the world arises from simple rules 28 .” These rules are generative rules, using feedback and learning algorithms to 
enable agents to adapt to their environments over the course of time. When these generative rules are applied to a 
population of agents, emergent behavior is observed. Hence, complexity science focuses on these generative rules to 
capture phenomena observed in the real world, with the eventual goal of enabling the understanding, prediction, and 
control of behavior in complex systems. While complexity science has largely targeted natural and biological 
systems, much of the resulting research on dynamical systems has relevance to the larger systems that exists in 
DLSCES when physics, organizations, and processes are considered simultaneously. A significant challenge for 
complexity science is calibrating the generative rules to the real data. Phelan provides an overview of challenges for 
complexity science, as well as successes. Laudan (1981) made the case that complexity science would remain 
progressive by solving problems and Lakatos (1974) stressed the need to demonstrate successful predictions. The 
field has seen such successes, such as Bak and Chen’s (1991) successful fit of power-law distributions to such real- 
world phenomena as sand piles, city sizes, and earthquakes. Significant challenges remain for the use in system 
dynamics, where calibration of models has been difficult 28 . In the context of interactions, many remaining research 
questions remain, with substantial opportunities for advancing the state of the art in LSCES. Research involving 
complexity science might be embedded within simulation capabilities that would inform MDO methods. Questions 
for research might include: what would be the generative rules that would allow simulation of emergent behavior 
within an organization, in the context of design of LSCES? How would such a simulation be incorporated into an 
MDO framework to enable a robust system design? Can these simulations be useful in developing meaningful 
uncertainty models pertaining to the human and organizational impacts on the system being designed? 

4. Organizational Perspectives 

Another aspect of inherent interactions during the design of LSCES are the interactions associated with the 
organization. These interactions exist at many levels including: organization to organization (e.g., between 
contractor and government agency); individual to individual in different parts of the same organization (e.g., 
between R&D and systems engineering or between engineering disciplines); and, at the highest level, organization 
to engineered system (e.g., between organizational incentives and engineered systems needs). The characteristics of 
each of these interactions may have significant impacts on various aspects of the systems development and design 
process as critical information is passed between people and groups including raw data and models, as well as less 
tangible but important concepts such as assumptions, experience, history, tacit knowledge, and clarifications of 
terminology. Hence, even with well-developed engineering interface methods, ineffective organizational interfaces 
may cause considerable inefficiencies during design. Several fields of study in the social sciences rigorously study 
organizational interactions and offer a rich area for further research in the design of LSCES 14 . Lor example, in 
addition to designing the engineered system, perhaps greater organizational efficiencies and performance may be 
obtained if we also strategically design organizational aspects including: incentives, teams, communication 
pathways, over-all organizational structure, contracting methods, and even office layouts. Blending fields such as 
business, organization science, psychology, sociology, and social network science with engineering strategies may 
ultimately lead to an improved design process. 

One example of related research are the studies on High Reliable Organizations (HROs). HROs are 
organizations that routinely and successfully operate high risk systems such as aircraft carriers, electric power grids, 
and air traffic control. The extensive research on HROs has led to the identification of several organizational 
strategies that increase reliability and reduce errors while operating complex, high risk systems such as LSCES. 
HRO research turns attention toward dealing with inevitable unknowns in working the LSCES, noting that “these 
systems due their complexity are formally underdetermined; that is, they are capable of assuming more conditions or 
system states than can be planned for or anticipated in formal designs. This means they have the capacity to confront 
managers with problems of high variety and significant novelty. 29 ” Correspondingly, HRO research suggests a 
sustained focus on organizational interactions noting that “reliability is not the outcome of organizational invariance, 
but, quite the contrary, results from a continuous management of fluctuations both in job performance and in overall 
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departmental interaction. It is the containment of these fluctuations, rather than their elimination, that promotes 
overall reliability. 30 ” 

It is also recognized by organization experts that the final system design depends inherently on the 
organizational structure in which it was realized. Organization theory provides numerous avenues for addressing 
these impacts and others in the Design for LSCES. 

D. Communication of System Objective(s) 

One of the most challenging needs in DLSCES is to enable all participants (sometimes as many as 10,000 plus) 
to understand how their decisions will impact the overall system objective. The present systems engineering 
approach results in a series of requirements being propagated from level to level in the hierarchy, where 
requirements result in a set of constraints given to the designers. In this way, the designer’s job becomes finding an 
alternative that fits within the set of constraints given to him by a higher level. There is less decision-making at this 
point, unless it is to decide that a requirement might need to be modified. 

One of the challenges with requirements is that they are not design independent - requirements are tied to 
solutions. In other words, once it is required that the lamp requires less than 100 Watt bulbs, you are certain to end 
up with a bulb-based lamp as your product rather than self-illuminating walls. If, however, one understands the 
system completely separate from possible solutions, then unique, innovative and truly optimal designs might be 
possible outcomes. 

One alternative to the present requirements flow-down approach is a strategy involving flow-down of a system 
objective. It has been suggested by proponents that this Value-Driven Design (VDD) approach 31, 32 would provide a 
clear value statement (objective) to which all participants in the process could then design. Such an objective would 
clearly express what is wanted, rather than what is not allowed by requirements. Another potential advantage of the 
VDD framework is the ability to more rapidly evaluate trades, not only across participating design teams, but also 
across all stages of the design and development process. The adoption of such a radically different approach would 
lead to fundamentally different design approaches. It would also require a significant change of our present 
acquisition system. Is it worth it? How can we know that such an approach would be worth the time and effort? 
Proponents of VDD suggest that failures in being able to effectively manage the requirements and couplings in these 
large-scale complex systems, together with the inability of all participants to understand the impact of their decisions 
in a systems context 31 , is reason enough that an investment in a new value-based approach might be appropriate. 

Value-Driven Design (VDD) 31 has been proposed as a framework against which system methods, processes, and 
tools can be assessed. Fundamentally, it is a framework that supports a value statement for the system (that captures 
the preferences appropriately) rather than requirements 33 . VDD professes to design for what is wanted, rather than 
what is not allowed. The VDD framework is used to form a system value model. The inputs of the system value 
model are the attributes of the system and of the system’s components. Some of the attributes that comprise the 
input set are physical attributes, such as weight and size, performance attributes, system costs and attributes that are 
commonly difficult to optimize in current design practices, such as maintainability, safety, schedule, risk and 
reliability. The model then outputs a singular value for a specific set of attributes. In this manner, designs with 
different attribute sets can be compared numerically. The VDD framework would also enable and facilitate the 
incorporation of manufacturing, operations, and disposal issues, enabling trade exploration across the lifetime of the 
system. The interacting roles of VDD, MDO and Decision Analysis (DA) are explored further in a systems context 
in the paper Collopy et al 34 . 

The hallmark of VDD lies in its ability to decompose the system value model into component objective 
functions 32 . By creating these objective functions, VDD eliminates the need for requirements on the attributes in the 
input set, allowing the true optimal design, according to the system value model, to be obtainable. Each component 
design team is supplied an objective function, replacing requirements such as “the height cannot exceed 2 feet,” 
which gives no preference information concerning designs. Each design team uses their objective function to 
determine the scalar values of different component designs. Using these scalar values the design teams are able to 
properly rank and determine the best design. 

The ability to flow down objective functions to the component level would certainly change the design 
environment. In VDD, systems engineers would be viewed as guiders of a system. In this model, systems engineers 
would flow down the component objective functions, monitor the system and component attributes, and modify 
objective functions in order to maintain balance in the system 31 . VDD impacts not only higher level systems 
engineers, but also design engineers. Designers and researchers of small components would not be quite as detached 
from how their decisions impact the system as a whole. By examining their objective function they could capture 
the value that one design has over another - not in terms of feasibility or infeasibility, but in terms of a scalar 
number. While it seems that the benefits of such an approach are potentially large, organizational implications must 
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also be addressed to ensure a VDD (or any approach) works efficiently. Organizational and social challenges such 
as ineffective leadership, poor (or lack of a) communication strategy, poorly written contracts and policies, 
incentives that conflict with system objectives, and interpersonal and inter-organizational relational problems, can 
grind design processes to a slow crawl if not addressed. Inefficiencies such as these may cause delays worth 
millions of dollars in developing a LSCES. 

Consider the example of the Marine’s Expeditionary Fighting Vehicle (EFV) program, intended to be able to 
deliver a crew of Marines from Navy ships to shore extremely quickly from 25 nautical miles away. The 
requirements were to achieve a 29 mph speed for the EFV, compared to the present Amphibious Assault Vehicles 8 
mph. The program was years behind in development and costs jumped from some $8 billion in 2000 to over $13 
billion in 2011, despite the fact that the number of vehicles to be purchased was cut in half. Experts questioned 
whether the stringent speed and distance requirements were valid. The Marine officials have since stated that it 
wasn’t really the EFV program itself that was important but rather the ability to storm beaches quickly at very high 
speeds. This is a case where a value statement from the outset might have been able to help the designers achieve 
the real objective without the burden of over-restrictive requirements 17,35 . 

Fields such as social network analysis can offer important insights regarding the quality and architecture of 
communication pathways within and external to the organization. In addition, the broad field of Positive 
Organizational Scholarship (POS) offers several strategies for building “positively deviant organizations” such as 
building and utilizing positive social capital through effective communication and networking approaches. Another 
example from POS that closely ties to communicating system objectives is open book management. This proven 
approach seeks to help every employee understand how their daily efforts tie to the company goals. Open book 
management and building positive social capital are two examples, of many business strategies, that potentially may 
be adopted, with adjustment, to engineering design to avoid organizational issues from impacting the design process, 
at a minimum. Looking forward, we hope that proactive integration of high performing organizational strategies 
with engineering strategies such as VDD may foster a leap forward in engineering design capabilities and 
efficiencies. However, there are numerous research questions yet to be addressed. Included among these are: Can 
value meaningful statements be obtained for LSCES? What would value statements for mission agencies look like 
as opposed to one in industry? What would be the resources necessary to determine a value statement that would 
adequately represent all aspects of the LSCES? Is it really values versus requirements or are there valuable hybrid 
approaches that could be developed? Can MDO be used as a framework to enable both a value statement and 
requirements? What would be the implications on procurement and acquisition procedures if such a framework were 
implemented? Would design participants have the flexibility and authority, within existing organizational structures, 
to make the trade decisions at the lowest levels, using a value-based framework? In what way would incentive 
structures be impacted by a value-driven approach? By an MDO hybrid approach? This topic provides numerous 
rich avenues of exploration. 

E. Decision-making and Sense-making with Ambiguity and Uncertainty in LSCES 

Uncertainty exists when an outcome cannot precisely be predicted. Uncertainty can have confounding effects on 
decision making and must therefore be understood and appropriately represented in the process so that decisions can 
be made in an informed way. Decision Analysis 36 provides a framework for thinking logically about the alternatives 
available and the choices to be made when there is uncertainty on the outcomes of those choices. 

Ambiguity 37 is just as critical a component, and pertains to a lack of clarity in which multiple interpretations can 
be made from the same information (or statements). Users of logic, including engineers, must ensure that there is no 
ambiguity in their activities. This becomes even more important in a systems context, wherein semantics within 
different groups (whether in the same organization or different organization) can lead to serious challenges 14 . 
Overcoming ambiguity requires communication amongst those participating in the process. The act of removing 
ambiguity leads to Sensemaking 38 ’ 39 , which deals with the cognitive gap that exists when individuals try to make 
sense of what they are seeing or hearing. 

Whereas uncertainty is an ignorance issue where we are lacking sufficient data to explicit questions, ambiguity is 
a confusion issue where we do not understand something and may not be sure what questions to ask. If people in a 
design team are confused about aspects of the system, then decision making is impaired. Consider Fig. 8 40 , in which 
a visualization of the Innolab 3D File Manager demonstrates the vast network of connections (and files) related to 
participants in a project. Each of these files essentially represents a product of a team that provides input to others. 
Without a structure or framework to facilitate determination and understanding of uncertainties and ambiguities, 
making decisions in such an environment is extremely challenging. 

Decision Analysis (DA) provides a means to facilitate decisions concerning allocation of resources in order to 
produce an embodiment of the preferences for the system, in the face of uncertainty. These decisions are based on 


15 

American Institute of Aeronautics and Astronautics 




Figure 8. Innolab 3D File Manager 


many things, including tacit knowledge and predictive modeling. Hence, when we make design decisions, there is a 
need to understand and appreciate the risks involved in a way that they can be designed for. 

As Howard 36 states, “Decision analysis is a logical procedure for the balancing of the factors that influence a 
decision. The procedure incorporates uncertainties, values, and preferences in a basic structure that models the 
decision.” DA builds upon a collection of methods drawn from multiple disciplines to provide a structured way to 
determine which alternative, amongst a set, an individual should choose. The alternatives that are being decided 
upon have uncertainties that affect the decision making process. 

Utility Theory (UT ) 41 is used in scenarios involving a single individual selecting from a set of alternatives. In 
UT, the alternatives are prescribed a value, referred to as its expected utility, dependent on the individual’s 
preferences and beliefs. The alternatives are ranked according to their expected utility and a choice is made. The 
preferences of an individual provide a mathematical representation of his or her desires concerning the alternatives 
and the degree by which they are risk averse or risk proverse. Uncertainties concerning an alternative are 
incorporated into UT through the individual’s beliefs on the alternative. These beliefs are typically represented 
through probability distributions. For example, an individual may have a belief concerning the accuracy of a 
particular design alternative’s performance result. This may affect the individual’s decision-making concerning the 
design alternative that they select. Through the use of Bayes Theorem 42 , an individual’s beliefs can be updated 
using new information through a mathematically proven means. In the previous example, the new information of 
the percent error of the performance result can be used to update the individual’s beliefs, affecting the choice they 
make. 

Game Theory (GT) 43, 44 is used to determine decisions that should be made in scenarios involving multiple 
interacting individuals. A “game” refers to the interactions that are occurring between decision-making individuals. 
Games occur many times over at any given instant, some being more subtle than others. For example, a game is 
being played in the situation involving a group of co-workers determining where to eat for lunch. A more obvious 
game is played during a professional sport’s draft where each team’s decisions affect the other team’s decisions. GT 
is essentially an extension of UT. The value each individual associates with each outcome of a game is determined 
through expected utility. Uncertainties involving other players of the game can be captured by the individual’s 
beliefs and incorporated into the game through the use of Bayesian Game Theory 45 . The solution of a game, 
referred to as a Nash equilibrium 46 , is defined as an outcome where each individual’s decision is a best response to 
every other individual’s decision. 

A critical component of the design process, particularly for the LSCES being discussed here, is the decision- 
making process that occurs by individuals across all levels of the organization. Design alternatives are presented 
and individuals or interacting individuals make a choice concerning them. In an organization, decision analysis can 
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be used to examine the micro-scale decision-making process involving a single design engineer responsible for a 
component, to the macro-scale decision-making process of games involving multiple contractors and sub-contractors 
interacting to produce one design outcome. The use of DA in engineering organizations, such as interactions 
between design teams in a company, have been explored and shown useful in the design process 47 . However, more 
research in the area of DA in design engineering is needed to further demonstrate its usefulness, as has been done 
previously in fields of economics, politics and military strategy 48 " 50 . 

An important topic in DA that needs further examination in the Design of LSCES is mechanism design. 
Mechanism design (MD) involves manipulation of a game, by its designer, to drive an outcome that he or she 
prefers 49 . Consider a typical case seen in design of a LSCES, in which numerous contractors each design separate 
components that will affect the performance of the other components. These components will ultimately need to 
come together to form the complete system. A game is played between the contractors to determine the 
characteristics of each component. The contractors make decisions based on their beliefs and preferences. In the 
context of Game Theory, then each contractor would choose the decision that is the best response to every other 
contractor’s decision, according to their own beliefs and preferences. The customer of the product has power in this 
game through mechanism design. Through designed incentives that can be flowed down to the contractors, the 
customer can drive an outcome that they prefer. This outcome may result in an outcome different than the one 
arrived at before the mechanism was enacted. Mechanism design is a powerful tool of DA that can alter the decision 
making processes that are the fundamental foundation of the design process. 

DA and the related tools of UT, GT, and MD offer a significant opportunity that may improve the LSCES design 
process. Though these methods provide a proven, rigorous approach, their effectiveness is a function of 
organizational dynamics. For example, as noted earlier, ambiguous information creates confusion that can greatly 
impair decision making. The inherently interdisciplinary nature of LSCES means that multiple disciplines and 
organizations, each with their own terminology and practices, must interact frequently during design, from R&D 
through production. The diversity of engineering language, organization culture, incentive systems, leadership, and, 
in a global design effort, diversity in native culture and language, all create an environment where conflicting data 
and information with multiple interpretations is common. Hence concepts such as DA are best applied along with 
organizational strategies. 

One such example is using sensemaking theory (which addresses issues related to ambiguity) along with DA. 
Sensemaking theory suggests that increased interpersonal interaction is still needed to address ambiguity and 
resulted confusions in the midst of our current information age. This theory also considers that, for many who work 
in complex environments, the main information related issue may not be a lack of information but rather the 
existence of too much information that has too many meanings. Adding to this may be disagreements between 
organizational entities (e.g., contractors, government agencies, departments, etc.) as well as new unknowns in the 
engineering system that arise during design. Hence, individuals engaged in engineering design must make decisions 
in the midst of significant challenges of ambiguity, organizational dynamics, and the persistent emergence of 
engineering unknowns and couplings in the engineered system. 

While DA and the associated tools of GT and MD are relatively well understood, they have not been investigated 
sufficiently in the organizational context of LSCES. Additionally, questions of how sensemaking and ambiguity 
interact and impact uncertainty and decision making are still very much open questions. Hence, there are numerous 
research challenges to be addressed. Questions include: can DA effectively be used for the scale of systems being 
considered? What degree of understanding and expertise would be necessary within an organization to enable 
effective use of DA? What may be some approaches or a framework that might support blending the theories of 
sensemaking (and other organizational theories) with DA? What might such a framework look like and how could it 
be effectively used within an organization? In what way would DA be used in an MDO framework? Would DA, GT 
and MD enable more effective simulation within an MDO framework? What is the relationship of generative rules 
being produced in complexity science to predict emergent behavior within the LSCES environment to the DA, GT 
and MD theories existing today? Are these compatible theories? If not, what does this mean and how would such 
incompatibility be explored? These are only a small subset of questions that would lead to valuable and insightful 
research in this critical area. 


V. Conclusion 

In this paper, we have described the challenges that presently exist in DLSCES, and identified some existing 
foundations from which a theory for DLSCES can be built. These include such valuable fields of study as Modeling 
and Simulation, Multidisciplinary Design Optimization (MDO), Complexity Science, Decision Analysis (including 
Game Theory and Mechanism Design), and many fields from the social sciences. We identified four (of many) 
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critical needs that could provide many rich areas of research opportunity, with many of them bridging technical and 
social fields. We attempted to identify opportunities that exist to produce paradigm-shifting research and far-ranging 
impact on the present and future practice in the Design of Large-scale Complex Engineered Systems. The authors 
have presented these needs and opportunities with the understanding that they represent our opinions and vision 
only. We recognize that the true challenge of advancing the design of LSCES will depend on the views and 
contributions of an extremely broad and diverse number of disciplines from technical and social fields. 
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